Method and system for searching for images based on color and shape of a selected image

ABSTRACT

A method and system that allows a user to find images using a query by example is disclosed. Images similar to the selected image are retrieved based on color or shape. Each clip in a clip catalog is assigned a color metric and a wavelet, i.e., shape, metric. The selected clip is also assigned a color metric and a wavelet metric. Those clips that are most similar based on either the wavelet metric or the color metric are returned to the user.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of patent application Ser. No.10/127,045, filed Apr. 19, 2002 now U.S. Pat. No. 6,944,332, and patentapplication Ser. No. 10/127,075, filed Apr. 19, 2002, now issued as U.S.Pat. No. 6,606,417 B1, both of which are divisionals of patentapplication Ser. No. 09/295,018, filed Apr. 20, 1999, now issued as U.S.Pat. No. 6,477,269 B1.

BACKGROUND

With today's technology, computer users have easy access to thousands ofdigital images. As technology continues to advance, more and morecomputer users will have access to more and more images. However, as thenumber of images to which computer users have access increases, so doesthe difficulty in locating a particular image.

Attempts have been made to solve this problem. One solution is the useof keywords. There are several difficulties with the use of keywords intrying to locate an image. First, all images must be manually tagged.Second, some visual aspects of an image are hard to describe. Third,users must guess which keyword(s) were used to tag images of potentialinterest.

Based on the problems with keyword searching, other approaches have beenused to describe images so that a database of images can be searched.One approach is to obtain a wavelet signature based on a waveletdecomposition to query by example. A wavelet signature describes theshape of objects located in an image First, the wavelet signature of aselected image that contains one or more features of interest to theuser (such as an image of an object) is determined. Then, waveletsignatures are determined for all potentially similar images that may beof interest to the user contained in an image catalog (i.e., an imagedatabase) to be searched. Next, the wavelet signatures of thepotentially similar images are compared with the wavelet signature ofthe selected image. The closer the wavelet signature of a potentiallysimilar image is to the wavelet signature of the selected image, thegreater the likelihood that the images are similar. Potentially similarimages are ranked using this criteria and the result displayed, inorder, to the user. There are some problems with this approach to imagesearching, for example, if an image is rotated, the original image andthe rotated image will produce significantly different waveletsignatures.

Another approach to image searching employs color histograms. A colorhistogram provides general information about the color characteristicsof an image. In this approach, the color histogram of a selected imagethat contains one or more features of interest to the user isdetermined. Then color histograms of all potentially similar imagescontained in an image catalog to be searched that may be of interest tothe user are determined. The color histograms are compared, ranked anddisplayed, in order, to the user. There are also problems with thisapproach, for example, a color, a black and white, and a gray scaleversion of the same image will produce significantly different colorhistograms. Accordingly, there exists a need for an easy method to moreaccurately locate images that are similar to a selected image.

SUMMARY

This summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This summary is not intended to identify key features ofthe claimed subject matter, nor is it intended to be used as an aid indetermining the scope of the claimed subject matter.

The present invention is directed to a system, method, andcomputer-readable medium for finding images in an image catalog that aresimilar to a selected image based on color or shape. Shape similarity isdetermined by comparing a wavelet signature derived from the selectedimage to a wavelet signature derived from each image in the imagecatalog. Color similarity is determined by comparing the color signatureof the selected image to the color signature of each image in the imagecatalog. A match value, or metric, is assigned for each of thesecomparisons. The images with the highest match value, i.e., closestmatch in either color or wavelet, i.e., shape, are returned to the user.

In accordance with other aspects of this invention, the waveletsignature is determined by: centering the image on a field; convertingthe image to a standard color space; normalizing the colors; performinga wavelet decomposition; determining coefficients for each color plane;and encoding the coefficients based on row/column and sign.

In accordance a further aspect of this invention, YIQ, a color modelused in broadcast television, is the standard color space used fordetermining the wavelet signature.

In accordance with yet a further aspect of the present invention, thecolors are normalized, preferably to vary from 0 to 1.

In accordance with still another aspect of this invention, the standardHaar transform is used for the wavelet decomposition.

In accordance with yet another aspect of this invention, 40 of thelargest coefficients are stored for each color plane. Preferably,coefficients below a threshold value are not stored.

In accordance with other aspects of the present invention, the colorsignature for an image is determined by: blurring the image; convertingthe image to a standard color space; accumulating color pixels inbuckets; computing pixel coverage for each of the color buckets; andencoding and storing coverage values for the colors with most coverage.

In accordance with further aspects of this invention, the color spaceused in determining the color signature is HSV (based on Hue, Saturationand Value elements).

In accordance with yet further aspects of this invention, coveragevalues are stored for the five colors with the most coverage. Preferablyonly colors with coverages above a threshold value are stored.

DESCRIPTION OF THE DRAWINGS

The foregoing aspects and many of the attendant advantages of thisinvention will become more readily appreciated as the same become betterunderstood by reference to the following detailed description, whentaken in conjunction with the accompanying drawings, wherein:

FIG. 1 is a block diagram of a general purpose computer system suitablefor implementing the present invention;

FIG. 2 is a sample Clip Gallery screen of results returned for a queryof clips similar to a selected clip based on color or shape;

FIG. 3 is a flow diagram illustrating the overall logic for computingand storing wavelet and color signatures for all images in an imagecatalog;

FIG. 4 is a flow diagram illustrating in detail the logic of computingand storing a wavelet signature for an image in accordance with thepresent invention;

FIGS. 5A–5I illustrate Haar transforms;

FIG. 6 is a diagram of shells and a shell numbering scheme;

FIG. 7 is an exemplary array of shell weight values for a YIQ colorspace;

FIG. 8 is a flow diagram illustrating in detail the logic for computingand storing a color signature for an image in accordance with thepresent invention;

FIG. 9 is a flow diagram illustrating the overall logic of a query forclips similar to a selected clip based on color or shape;

FIG. 10 is a flow diagram illustrating in detail the logic of searchingan image catalog for images similar to a selected image;

FIG. 11 is a flow diagram illustrating in detail the logic ofdetermining a wavelet metric;

FIG. 12 is a flow diagram illustrating in detail the logic ofdetermining a color metric;

FIG. 13 is a graph of wavelet and color metrics;

FIGS. 14A–14C illustrate exemplary wavelet signatures and metrics for aselected image and two target images; and

FIGS. 15A–15C illustrate exemplary color signatures and metrics for aselected image and two target images.

DETAILED DESCRIPTION

The present invention is directed to a computer program for searching aclip catalog (i.e., a database) containing a plurality of image clips tofind images that are similar to a selected image based on color and/orshape. The invention is a search tool designed for use on a computer,such as a personal computer or a network computer system running amultimedia application program that contains a catalog of clips, such asthe Clip Gallery produced by Microsoft Corporation, Redmond, Wash.

Exemplary Computing Environment

FIG. 1 and the following discussion are intended to provide a brief,general description of a suitable computing environment in which thepresent invention may be implemented. Although not required, theinvention will be described in the general context ofcomputer-executable instructions, such as program modules, beingexecuted by a personal computer. Generally, program modules includeroutines, programs, characters, components, data structures, etc., thatperform particular tasks or implement particular abstract data types. Asthose skilled in the art will appreciate, the invention may be practicedwith other computer system configurations, including hand-held devices,multiprocessor systems, microprocessor-based or programmable consumerelectronics, network PCs, minicomputers, mainframe computers, and thelike. The invention may also be practiced in distributed computingenvironments where tasks are performed by remote processing devices thatare linked through a communications network. In a distributed computingenvironment, program modules may be located in both local and remotememory storage devices.

With reference to FIG. 1, an exemplary system for implementing theinvention includes a general purpose computing device in the form of aconventional personal computer 20, including a processing unit 21,system memory 22, and a system bus 23 that couples various systemcomponents including the system memory 22 to the processing unit 21. Thesystem bus 23 may be any of several types of bus structures including amemory bus or memory controller, a peripheral bus, and a local bus usingany of a variety of bus architectures. The system memory includesread-only memory (ROM) 24 and random access memory (RAM) 25. A basicinput/output system (BIOS) 26, containing the basic routines that helpsto transfer information between elements within the personal computer20, such as during start-up, is stored in ROM 24. The personal computer20 further includes a hard disk drive 27 for reading from and writing toa hard disk 39, a magnetic disk drive 28 for reading from or writing toa removable magnetic disk 29, and an optical disk drive 30 for readingfrom or writing to a removable optical disk 31, such as a CD-ROM orother optical media. The hard disk drive 27, magnetic disk drive 28, andoptical disk drive 30 are connected to the system bus 23 by a hard diskdrive interface 32, a magnetic disk drive interface 33, and an opticaldrive interface 34, respectively. The drives and their associatedcomputer-readable media provide non-volatile storage ofcomputer-readable instructions, data structures, program modules, andother data for the personal computer 20. Although the exemplaryenvironment described herein employs a hard disk 39, a removablemagnetic disk 29, and a removable optical disk 31, it should beappreciated by those skilled in the art that other types ofcomputer-readable media which can store data that is accessible by acomputer, such as magnetic cassettes, flash memory cards, digital videodisks, Bernoulli cartridges, random access memories (RAMs), read-onlymemories (ROMs), and the like, may also be used in the exemplaryoperating environment.

A number of program modules may be stored on the hard disk 39, magneticdisk 29, optical disk 31, ROM 24 or RAM 25, including an operatingsystem 35, one or more application programs 36, other program modules 37and program data 38. A user may enter commands and information into thepersonal computer 20 through input devices such as a keyboard 40 andpointing device 42. Other input devices (not shown) may include amicrophone, joystick, game pad, satellite dish, scanner, or the like.These and other input devices are often connected to the processing unit21 through a serial port interface 46 that is coupled to the system bus23, but may also be connected by other interfaces, such as a parallelport, game port or a universal serial bus (USB). A display in the formof a monitor 47 is also connected to the system bus 23 via an interface,such as a video card or adapter 48. Preferably, the video card oradapter includes a 3D rendering engine implemented in hardware. One ormore speakers 57 may also be connected to the system bus 23 via aninterface, such as an audio adapter 56. In addition to the display andspeakers, personal computers typically include other peripheral outputdevices (not shown), such as printers.

The personal computer 20 may operate in a networked environment usinglogical connections to one or more personal computers, such as a remotecomputer 49. The remote computer 49 may be another personal computer, aserver, a router, a network PC, a peer device or other common networknode, and typically includes many or all of the elements described aboverelative to the personal computer 20. The logical connections depictedin FIG. 1 include a local area network (LAN) 51 and a wide area network(WAN) 52. Such networking environments are commonplace in offices,enterprise-wide computer networks, intranets, and the Internet.

When used in a LAN networking environment, the personal computer 20 isconnected to the local area network 51 through a network interface oradapter 53. When used in a WAN networking environment, the personalcomputer 20 typically includes a modem 54 or other means forestablishing communications over the wide area network 52, such as theInternet. The modem 54, which may be internal or external, is connectedto the system bus 23 via the serial port interface 46. In a networkedenvironment, program modules depicted relative to the personal computer20 or portions thereof may be stored in the remote memory storagedevice. It will be appreciated that the network connections shown areexemplary, and other means of establishing a communications link betweenthe computers may be used.

The present invention, implemented on a system of the type illustratedin FIG. 1 and described above, allows a user to efficiently andeffectively search a clip catalog to locate the clip or clips that mostclosely match what the user is seeking. As will be better understoodfrom the following description, this is accomplished by a user selectinga clip, and invoking a command to find clips of similar color or shape.

A Clip Gallery user interface 100 is illustrated in FIG. 2. The userinterface 100 represents the display returned to a user after invoking acommand to find. clips similar in color and shape to a selected clip102. The display shows the return of resulting clips 104–116.

Determining Wavelet and Color Signatures

FIG. 3 is a flow diagram illustrating the overall logic for computingand storing wavelet signatures and color signatures for all of theimages in an image catalog. The logic moves from a start block to block116 where standard color space weight values, preferably YIQ, are storedin an array for fast retrieval. This array is a table of weights foreach color channel (‘Y’, ‘I’, and ‘Q’) for each shell, and isillustrated in FIG. 7. The logic then moves to decision block 120 wherea test is made to determine if there are more images. If so, the logicmoves to block 122 where the next image in the image catalog isobtained. The logic then moves to block 124 where a wavelet signature iscomputed and stored for the image. The logic for computing and storing awavelet signature for an image is illustrated in detail in FIG. 4 anddescribed below. Next, in block 126, a color signature is computed andstored for the image. The logic for computing and storing a colorsignature for the image is illustrated in detail in FIG. 8 and describedlater. The logic then returns to decision block 120. The logic of blocks120–126 is repeated until there are no more images in the image catalogthat need to have wavelet and color signatures computed, at which pointthe logic of FIG. 3 ends. While the logic illustrated in FIG. 3 can beperformed at any time in order to compute wavelet and color signaturevalues for all of the images stored in an image catalog, alternatively,the wavelet and color signatures can be computed whenever an image isadded to the clip catalog and whenever an image in the clip catalog ismodified and the wavelet and color signatures stored for subsequent usein the manner described below.

Determining Wavelet Signatures

A wavelet signature is derived from a wavelet decomposition. Thecoefficients of the decomposition are stored into small signatures foreach image. An image querying metric is computed from these signatures.The metric essentially compares how many significant waveletcoefficients the query has in common with potential target images. Thismetric requires very little storage for the signatures of the images.Comparisons of the metric are fast, thereby allowing for a rapid searchof a database containing thousands of images.

The logic shown in FIG. 4 for computing a wavelet signature for an imagemoves from a start block to block 130 where the image is centered on afield, preferably white. Preferably, the field is 128×128 pixels. A128×128 pixel field will accommodate thumbnails, which arerepresentations of images, such as the ones that are displayed in theClip Gallery user interface shown in FIG. 2. Still referring to FIG. 4,the logic next moves to block 132 where the image is converted to astandard color space, preferably YIQ. YIQ is a standard color model usedby broadcast television based on brightness, color and tint. Thisstandard is the preferred model because it most closely matches a user'sperception of an image. Other color space standards, such as RGB (Red,Green, Blue) or HSVA (Hue, Saturation, Luminosity, Alpha) can also beused. Algorithms for converting an image to a standard color space, suchas YIQ or RGB, are known to those skilled in the art of computergraphics, particularly with respect to the image processing aspect ofcomputer graphics.

Returning to FIG. 4, colors are normalized by rescaling, preferably, tovary from values between 0 and 1. See block 134. Next, the logic movesto block 136 where a wavelet transform is performed on the normalizedvalues. Preferably, the standard Haar wavelet transform is used becauseit is simple to implement and provides for fast computations. Othertransforms may be used, and may even provide more accurate results,however, they will most likely be slower. The Haar wavelet transform isillustrated in detail in FIGS. 5A–5I and is described in detail later.

After performing the wavelet transform, the logic moves to block 138where a specified number, X, of the largest coefficients by magnitudefor each color plane are found. Preferably, the number of coefficients,X, is forty. For a 128×128 pixel image, there are 128² or 16,384different wavelet coefficients for each color channel (‘Y’, ‘I’, and‘Q’). It is preferable that the sequence be truncated with only thecoefficients with the largest magnitude being used. This truncationaccelerates the search and actually appears to improve the resultsbecause only the most significant features are considered, i.e., “noise”is removed. A truncated value of about 40 coefficients provides thepreferred results in both accuracy and efficiency. Preferably, valuesfalling below a threshold value are ignored. Therefore, at most fortycoefficients will be found for each color channel. However, less thanforty coefficients may be found because there may not be fortycoefficients above the threshold value.

The logic next moves to block 140 where row/column locations ofcoefficients are encoded by increasing resolution. This is done usingshells where each shell represents a resolution. FIG. 6 illustrates ascheme for ordering shells based on resolution, and is described indetail later. The signs of the coefficients are then encoded, preferablyusing the high order bit. The logic then moves to block 144 where thespecified number, X, of the largest encoded coefficients are stored insorted order. Using this scheme, only 240 bytes of storage are requiredto store the coefficient data for an image (40 coefficients for 3 colorchannels, with each coefficient requiring 2 bytes of storage, including14 data bits and 1 sign bit). Next, in block 146, the wavelet signatureis stored. The logic of FIG. 4 then ends, and processing is returned toFIG. 3.

Standard Haar Transform

Even though there are an infinite number of wavelet transforms, thereare properties common to all wavelet transforms. Wavelet transformsdecompose input signals (functions, images, etc.) into “average” and“difference” components. FIGS. 5A–5I illustrate how this is accomplishedusing the Haar transform.

Simply stated, the Haar transform uses two numbers, a and b. The averageis defined as avg=(a+b)/2 and the difference is defined as diff=(a−b).For these numbers (avg and diff) to be useful, a capability must existthat allows for reconstruction of the original numbers from them. Thisreconstruction is possible because a=avg+(diff/2) and b=avg−(diff/2). Inthe Haar transform the average is defined as avg=(a+b)/sqrt(2) and thedifference is defined as diff=(a−b)/sqrt(2), where sqrt(2) which is thesquare root of 2 or 1.41421, and is used to normalize the values.Defined this way the inverse process (the reconstruction of the originalnumbers) can be calculated in virtually the same way;a=(avg+diff)/sqrt(2) and b=(avg−diff)/sqrt(2). This type ofcomputational symmetry is called orthogonality and is an importantproperty shared by all wavelets.

An example, shown in FIGS. 5A and 5B, will help clarify thesedefinitions. FIG. 5A illustrates the forward Haar transform on an inputsignal (or function). The forward Haar transform is defined by theaverage components determined by equation 1 and the differencecomponents determined by equation 2:H _(i)=(A _(2*i) +A _(2*i+1))/sqrt(2)  (1)where i=0 . . . (N/2)−1.H _((N/2)+1)=(A _(2*i) −A _(2*i+1))/sqrt(2)  (2)where i=0 . . . (N/2)−1.

The inverse Haar transform is defined by the even components determinedby equation 3 and the odd components determined by equation 4:a _(2*i)=(H _(i) +H _(i+(N/2)))/sqrt(2)  (3)where i=0 . . . (N/2)−1.a _(2*i+1)=(H _(i) −H _(i +(N/2)))/sqrt(2)  (4)where i=0 . . . (N/2)−1.

In the example illustrated in FIGS. 5A and 5B, the Haar transform wasrun once (or one level) over the input signal. The difference componentof the first level can be used to show where dramatic changes in theinput signal occur. A large difference indicates a large change betweeninput signal neighbors. Values near zero are indicative of input signalneighbors that are close to the same value and somehow similar. Byrunning the Haar transform again on the first level average componentschange information can be generated about a larger part of the signalsince the difference of two averages contains information about 4 inputsignal values. This process can be repeated until the average anddifference components contain one element each. Each successivetransform level provides courser frequency (change) information about alarger part of the input signal. To reconstruct the signal from thetransform, the average and difference components of the coarsest levelare used to generate the next coarsest average. The reconstructedaverage and its corresponding difference components are then used toreconstruct the next average. This is continued until the originalsignal is regenerated. The table shown in FIG. 5C shows the completeHaar transform for the example above.

Since average components can be reconstructed for a given level from thenext coarsest level, only the difference components and the last level'saverage components need to be saved in order to reconstruct the entireinput signal. FIG. 5C illustrates a natural ordering of the Haartransform data. This ordering not only makes it easy to find where alevel begins, it also saves computer memory, which is an importantconsideration when computing the wavelet transform of a large image.This ordering is illustrated in FIG. 5D. The graphs shown in FIGS. 5E–5Isummarize the discussion above.

FIG. 6 is a diagram illustrating shells used to store the encoded valuesof the highest coefficients. See block 144 of FIG. 4. FIGS. 14A–14C,described later, illustrate examples of coefficient values stored in ashell matrix, such as the one shown in FIG. 6. Referring to FIG. 6, thefirst shell, “Shell 0,” 300 contains the average color coefficient forthe image, and is surrounded by the next shell, “Shell 1,” which iscomposed of three coefficients, 302, 304, and 306. The next shell,“Shell 2” surrounds “Shell 1”. In this manner, each shell surrounds theprevious shell. Each shell represents the next level of resolution, andis assigned a weight value. Preferable shell weight values are shown inFIG. 7. The weight values, as shown in FIG. 7, are stored in an array(block 116 of FIG. 3). The coefficient of “Shell 0” 300 is not weighted,i.e., has a weight value of 0. “Shell 0” 300 is ignored because itrepresents color averages not shape. For a 128×128 pixel image, theshells as shown in FIG. 6 would be in a 128×128 matrix. All coefficientvalues that are not among the largest coefficients, e g., the largestforty coefficients, are assumed to be zero. Actual coefficient valuesmay be maintained, or the largest (absolute value) coefficients may bestored as +1 or −1. The shell locations for the largest fortycoefficients are sorted and stored in an array. For example, if thelargest coefficient were stored in cell number two (304 in “Shell one”),a two (for cell number 2) would be stored in the array, along with thecoefficient value (or +1 or −1). The high order bit is used to indicatewhether the coefficient is a negative value. By storing the top 40coefficients in a sorted array, a linear comparison can be made betweenthe selected image and a target image, thereby minimizing thecomputation time required for comparing images for similarity.

Determining Color Signatures

FIG. 8 is a flow diagram illustrating in detail the logic of computing acolor signature for an image (block 126 of FIG. 3). The logic moves froma start block to block 400 where the image is blurred to removedithering artifacts. Preferably, a 3×3 kernel is used to blur the image.Next, in block 402, the image is converted to a standard color space,preferably, HSV (based on Hue, Saturation and Value elements). It willbe appreciated that a different color space can be used, for example,RGB (Red, Green, Blue), or HSB (Hue, Saturation, and Blacknesselements). The logic then moves to block 404 where pixels areaccumulated in color buckets. Preferably the number of color buckets, Y,matches the number of colors in a color identification table. Forexample, a color identification table may contain the colors: red,orange, yellow, yellow-green, green, green-cyan, cyan, cyan-blue, blue,blue-purple, purple, pink, brown, white, gray, black, and transparent.There should be 17 color buckets for the color identification tabledescribed above, one bucket for each entry in the color identificationtable. Another example of a color identification table contains theentries: light red, dark red, light orange, dark orange, light yellow,dark yellow, light green, dark green, light cyan, dark cyan, light blue,dark blue, light, purple, dark purple, white, gray, black, andtransparent. There should be 18 color buckets for the example justdescribed. Using a color identification table, in order for a pixel tobe counted, it must be an exact match. This allows for fastcomputations, however, it may not produce the best results. An extremecase is the comparison of an all red image to an all orange image. Whilethese two images are visually similar they will not be a match. Toovercome this problem, an alternative embodiment may use a color crosscorrelation matrix which specifies weighted values, with high weightsfor exact color matches, and lower weights for similar colors, such asred and orange. Still referring to FIG. 8 the percentage of pixelscovered by each bucket is computed. See block 408. A specified number ofbuckets, Z, with the largest percentage of coverage are then found. Seeblock 410. Five is a preferable number of buckets because it allows for1 background color, 1 foreground color and 3 other colors. Preferably,values falling below a specific threshold are ignored. Next, in block412, bucket numbers and coverage values are encoded, preferably withvalues ranging from 0 to 255, with 255 representing 100% coverage. Othervalues, such as a percentage value (0 to 100) could be used. The logicthen moves to block 414 where an array of the specified number ofbuckets, Z, with encoded values for the greatest coverage are stored ina sorted list. The array is 10 bytes, 1 byte for a color value (i.e., acolor bucket number), and another byte for the coverage value, for thetop five colors. The logic then moves to block 416 where the colorsignature is stored. The logic of FIG. 8 then ends and processingreturns to FIG. 3.

Finding Similar Images Based on Wavelet and Color Signatures

FIG. 9 is a flow diagram illustrating the overall logic of findingimages most similar to a selected image, based on the color and/or shapeof a selected image. First, in block 500, a user selects an image,preferably via a suitable user interface, for example using the ClipGallery user interface. Next, in block 502, a search is performed tofind images in a clip catalog that are similar to the selected image.The logic of searching for similar images 502 is illustrated in detailin FIG. 10, and described next. After searching for similar images, thelogic moves to decision block 504 where a test is made to determine ifanother search should be performed. If so, the logic returns to block500. The logic of blocks 500–504 is repeated until it is determined thatanother search should not be performed, at which point the logic of FIG.9 of finding similar images ends.

The logic of FIG. 10 of searching for similar images based on colorand/or shape moves from a start block to block 520 where the totalwavelet value is computed for the selected image. The total waveletvalue is computed by summing the weights for the highest, i.e., stored,coefficients. This is clarified by the example illustrated in FIGS.14A–14C, and described in detail below. The logic of FIG. 10 then movesto block 522 where a target image is retrieved to compare to theselected image. A wavelet metric is determined. See block 524. FIG. 11illustrates in detail the logic of determining a wavelet metric. Next,in block 526, a color metric is determined. The logic of determining acolor metric is illustrated in detail in FIG. 12. The logic then movesto decision block 528 to determine if there are more images. If so, thelogic returns to block 522, and the logic of blocks 522–528 is repeateduntil it is determined in block 528 that there are no more images. Whenthere are no more images, the logic moves to block 530 where images aresorted by the larger of the wavelet metric and the color metric. Theimages with the largest metric, either wavelet or color, are returned tothe user. Preferably, a specified number of images with the largestmetric are returned to the user, for example 60. Alternatively, imageswith metrics over a threshold value could be returned, or a specifiedamount, such as 60 could be returned, but only if the metric exceeded athreshold value, i.e., at most 60 images are returned. The logic of FIG.10 then ends, and processing is returned to FIG. 9. While the preferredembodiment illustrated and described selects images with the highestvalue, either based on color or wavelet, i.e., shape, alternativeembodiments could select images based on a combination of color andshape, such as by summing or averaging the color metric and the waveletmetric.

FIG. 11 illustrates in detail the logic of determining a wavelet metric(block 524 of FIG. 10). The wavelet metric is a value indicating howclosely a target image matches the selected image based on the shapes ofthe images. The values range from 0 to 1 with 1 being an exact match.The logic moves from a start block to block 540 where a total waveletvalue for the target image is computed by summing the weight values ofthe target image coefficients. The logic next moves to block 542 wherethe wavelet signature for the selected image is compared to the waveletsignature for the target image. This is done by finding all matchingencoded values in the wavelet signature. When a match is found, theweight value is retrieved from the stored array shown in FIG. 7. Thewavelet weights for matches are then summed, in block 544. Next, inblock 546 a wavelet metric is determined by dividing the wavelet weightsum computed in block 544 by the larger of the two total wavelet valuesfor the selected image and the target image. The logic of FIG. 11 thenends, and processing returns to FIG. 10.

FIG. 12 illustrates in detail the logic of determining a color metric(block 526 of FIG. 10). The color metric provides an indication of howclosely a target image matches the selected image based on color. Themetric value ranges from 0 to 1, with 1 being an exact match. The rangevalues parallel the values for the wavelet metric. This allows for thelarger values to be returned to the user, regardless of which metric wasused. In this manner, the user is presented with images which mostclosely match the selected image based on either color or shape. Thelogic moves from a start block to block 550 where minimum coveragemultiplied by the color weight for all pairs of colors in the two imagesis summed. If a color identification table is used, the weight for anexact match is 1, and non-matching colors have a weight value of 0. If acolor cross correlation matrix is used, exact matches and similarmatches will have weight values greater than 0, but less than 1 withhigher weight values for closer matches. For a given color, the sum ofthe weight values should equal 1. The color weight sum is then raised toa power, preferably four, to normalize the color weight. This is done tonormalize the color metric to more closely resemble the closeness valuesfor shape (wavelet metrics), and is shown in FIG. 13 and described next.The logic of FIG. 12 then ends, and processing returns to FIG. 10.

FIG. 13 is a graph illustrating raising an image to the power of four(block 532 of FIG. 10). The dotted line represents the color metricbefore exponentiation. The solid line represents the wavelet metric. Thesolid line also approximates the color metric after exponentiation. Ascan be seen, the exponentiation changes the color scale to be similar tothe wavelet metric.

An Illustrated Example

FIGS. 14A–14C provide an illustrated example of determining waveletmetrics for a selected image and two target images. The illustrateexample shows only the ‘Y’ coefficients. Similar matrices are requiredfor ‘I’ coefficients and ‘Q’ coefficients. FIG. 14A represents a simpleexample of a matrix containing the largest coefficients for the selectedimage. The coefficients in the illustration are either +1, −1 or 0(blank). As described earlier, actual coefficient values can be stored,or a +1 or −1 to indicate a large coefficient, either positive ornegative, i.e., “large” is based on the absolute value of thecoefficient, can be used (FIGS. 14A–14C use the latter method). Todetermine the total wavelet value for the selected image shown in FIG.14A, the weight values for each of the coefficients are summed. Sincethere is one coefficient, +1, in Shell 1, the weight for Shell 1 isretrieved from the stored array. The coefficients illustrated are storedin block 144 of FIG. 4, and the weight values are stored in block 116 ofFIG. 3. The weight values for the coefficients are shown in FIG. 7. Thisprocess is repeated for all of the stored coefficients, 40 coefficientsfor ‘Y’, 40 coefficients for ‘I’ and 40 coefficients for ‘Q’. In theillustrated example of the ‘Y’ coefficients, there is one coefficient inShell 1, one coefficient in Shell 2, one coefficient in Shell 3, and onecoefficient in Shell 5. Using the weight values of FIG. 7, this yields atotal wavelet value of 2660 (830 for the ‘Y’ coefficient in Shell 1,1010 for the ‘Y’ coefficient in Shell 2, 520 for the ‘Y’ coefficient inShell 3, and 300 for the ‘Y’ coefficient in Shell 5). FIGS. 14B and 14Crepresent target images. A total wavelet value is computed for eachtarget image in the same manner as the selected image (block 540 of FIG.11). Each target image can be compared to the selected image bydetermining if there is a match for a given element of the matrix. Thematch must be exact, i.e., positive coefficient or negative coefficient,i.e., a+1 or −1 for the same element of both the target image matrix andthe selected image matrix. Matching elements are underlined in FIGS. 4Band 4C. The matching elements are summed, for example, the sum is 1130in FIG. 4B and 2360 in FIG. 4C. A match value is then determined bydividing the sum of the matching elements by the larger total waveletvalue (selected image or target image). The values (rounded to thenearest thousandth) for the examples illustrated in FIGS. 4B and 4C areunderlined and are 0.359 and 0.700, respectively. The shape of the imageof FIG. 4C matches the selected image more closely than the image ofFIG. 4B.

FIGS. 15A–15C provide an illustrated example of determining colormetrics for a selected image and two target images. FIG. 15A shows the 5colors with the greatest coverage scaled to values between 0 and 255(i.e., 255 equals 100% coverage), for the selected image. The values arestored in a ten byte array containing a color bucket index and acoverage value, for example, the values shown in FIG. 15A would bestored as {0, 60, 8, 40, 12, 30, 13, 75, 16, 50}. See FIG. 8, block 414.FIGS. 15B and 15C show color coverages for two target images. For eachtarget image, the stored colors with the greatest coverage are comparedto the stored colors for the selected image. If there is a match, thesmaller coverage value is multiplied by the color weight value. If acolor identification table is used, only exact matches will be counted,and the weight value for an exact match is always 1. Alternatively, if acolor cross correlation matrix is used, if there is a match, the matchcan be either exact color or similar color, and the weight value for amatch is greater than 0 and less than 1. For the target imageillustrated in FIG. 15B, these values add up to 195. This value is thennormalized by dividing it by 255. The value is then raised to the fourthpower so that the color metric more closely models the wavelet metric,as shown in FIG. 13. The resulting values (rounded to the nearestten-thousandth) are underlined in the examples illustrated in FIGS. 15Band 15C, and are 0.3420 and 0.4176, respectively. Therefore, the targetimage of FIG. 15C is more similar in color to the selected image thanthe target image of FIG. 15B. The illustrated example uses a coloridentity table, therefore only exact matches are used, with an exactmatch having a weight value of 1. If a color cross correlation matrix isused similar matches would also be included, but exact matches andsimilar matches would have weight values of less than 1. For example, inFIG. 15B, in addition to exact matches, similar matches, such as ayellow and a brown match, a brown and red match might also be included.The weight values of the similar matches should be less than those forexact matches. For example, a yellow and brown match may have a weightvalue of 0.2, whereas an exact brown match may have a weight value of0.4. The total weight value for all matches for a given color, i.e.,exact matches and similar matches must total 1.

In the illustrated example, the wavelet metric for target image 1 is0.359 and the color metric is 0.3420. The wavelet metric for targetimage 2 is 0.700 and the color metric is 0.4176. The wavelet metric islarger than the color metric for both of the target images, therefore,the wavelet metric will be used in both cases in determining whether theimage should be returned to the user. See FIG. 10, block 530. Asmentioned previously, in alternative embodiments, the two metrics(wavelet and color) could be summed or averaged in order to returnimages that are most similar in both color and shape.

While illustrative embodiments have been illustrated and described, itwill be appreciated that various changes can be made therein withoutdeparting from the spirit and scope of the invention.

1. A computer-implemented method for finding images, comprising:computing a first set of signatures that describes shapes in a selectedimage, the first set of signatures being a set of largest coefficientsof a wavelet transform for each color plane of the selected imagewithout calculating types of quantized coefficients; computing a secondsignature that describes colors in the selected image; and findingimages that are similar to the selected image using the first set ofsignatures and the second signature.
 2. The method of claim 1, furthercomprising storing standard color space weight values in an array forretrieval, the array being a table of weights for each color channel. 3.The method of claim 2, wherein the act of computing the first set ofsignatures includes computing a wavelet signature and storing thewavelet signature for the selected image.
 4. The method of claim 1,wherein the acts of computing the first set of signatures and the secondsignature are executed at any time for all images stored in an imagecatalog.
 5. The method of claim 1, wherein the acts of computing thefirst set of signatures and the second signature occur whenever an imageis added to a clip catalog.
 6. The method of claim 1, wherein the actsof computing the first set of signatures and the second signature occurwhenever an image in a clip catalog is modified.
 7. A computer-readablemedium having computer-executable instructions stored thereon forimplementing a method for finding images, comprising: computing a firstsignature that describes shapes in a selected image without computingquantized coefficients for each subband separately; computing a secondsignature that describes colors in the selected image using a colorcross correlation matrix which specifies weighted values; and findingimages that are similar to the selected image using the first signatureand the second signature.
 8. The computer-readable medium of claim 7,further comprising storing standard color space weight values in anarray for retrieval, the array being a table of weights for each colorchannel.
 9. The computer-readable medium of claim 8, wherein the act ofcomputing the first signature includes computing a wavelet signature andstoring the wavelet signature for the selected image.
 10. Thecomputer-readable medium of claim 7, wherein the acts of computing thefirst signature and the second signature are executed at any time forall images stored in an image catalog.
 11. The computer-readable mediumof claim 7, wherein the acts of computing the first signature and thesecond signature occur whenever an image is added to a clip catalog. 12.The computer-readable medium of claim 7, wherein the acts of computingthe first signature and the second signature occur whenever an image ina clip catalog is modified.
 13. A computer-implemented method forfinding images, comprising: computing a wavelet decomposition thatcontains coefficients for a selected image without using Daubechiesorthonormal wavelet filter to obtain subbands; deriving signatures thatdescribe shapes of the selected image from the coefficients; computingan image querying metric from the signatures by truncating thecoefficients to retain those with the largest magnitude for each colorchannel; and comparing a number of coefficients a query has in commonwith target images to find desired images by storing the number ofcoefficients in a sorted array and linearly comparing the number ofcoefficients.
 14. The method of claim 13, wherein the act of computingthe wavelet decomposition includes centering the selected image on afield, the field being white and its dimensions being about 128 pixelsby 128 pixels.
 15. The method of claim 14, wherein the act of computingthe wavelet decomposition includes converting the selected image to astandard color space, wherein the standard color space includes a colorspace uses by broadcast television.
 16. The method of claim 15, whereinthe act of computing the wavelet decomposition includes rescaling thecolors to produce normalized values.
 17. The method of claim 16, whereinthe act of computing the wavelet decomposition includes performing awavelet transform on the normalized values.
 18. The method of claim 17,wherein the act of computing the wavelet decomposition includesdetermining the number of the largest coefficients by using magnitudefor each color plane.
 19. The method of claim 18, wherein the act ofcomputing the wavelet decomposition includes truncating a sequence ofcoefficients to retain coefficients with the largest magnitude.
 20. Themethod of claim 19, wherein the act of computing the waveletdecomposition includes encoding row and column locations of coefficientsby increasing the resolution.